Cross-Cultural Adaptability 1 Running Head: Cross-Cultural Adaptability Inventory Examining the Psychometric Properties of the Cross-Cultural Adaptability Inventory

The current study examined the factor structure of the Cross-Cultural Adaptability Inventory (CCAI) to better understand the psychometric properties of this instrument. Subsequent to the current study, the CCAI, which was developed to provide a selfinventory of an individual’s cultural adaptability, had limited study of its measurement characteristics. Therefore, the structure was examined using data collected from a group of university students. Confirmatory factor analysis indicated poor fit of the four-factor model proposed by the test authors. Follow-up exploratory factor analyses failed to reveal an interpretable structure. Possible explanations for poor fit are discussed and recommendations for further research are suggested. Cross-Cultural Adaptability 3 Examining the Psychometric Properties of the Cross-Cultural Adaptability Inventory The Cross-Cultural Adaptability Inventory (CCAI) was developed to provide a tool for self-assessment of cross-cultural effectiveness. This instrument, which was originally created in 1987, was revised in both 1989 and 1992. The instrument’s authors, Colleen Kelley and Judith Meyers, created both the original and the revised versions of the instrument. Designed to be used as a single assessment or as part of a multiassessment training program, the CCAI was developed in response to the need for a selfassessment instrument designed to measure cross-cultural adaptability (Kelley & Meyers, 1999). The authors stated that this instrument is applicable to all cultures assuming that anyone who was adapting to a new culture would share the same types of feelings and experiences (Kelley & Meyers, 1995a). The manual for the CCAI (Kelley & Meyers, 1995a) presents a limited description of the history of the development of the instrument. Specifically, after a review of the literature, the authors created a Cross-Cultural Readiness checklist of the characteristics cited in the literature as being important for cross-cultural adaptability. A panel of experts then rated the significance of each characteristic on the checklist in respect to adapting to other cultures. The characteristics with the highest ratings were then grouped into four categories (flexibility/openness, emotional resilience, perceptual acuity, and personal autonomy). Based on information from the cross-cultural adaptability literature, the authors then added a fifth category (positive regard for others). Ten items were then written to represent each of the five categories. Using cross-cultural experts and members of the general public, feedback pertaining to the CCAI items was gathered and used to make revisions. This version of the instrument (1987) was administered to obtain norming data. The normative sample consisted of 653 individuals with a variety of different occupations, levels of education, and age groups. The sample was approximately 63% male and 80% of the participants were citizens of the United States with the other 20% representing other countries from around the world. Detailed statistics about the normative sample are presented in the manual for the CCAI (Kelley & Meyers, 1995a). Using the results from this administration, the items previously representing the category Positive Regard for Others were reassigned to the remaining four categories creating the 1989 version of the instrument. The authors then state that in 1991 “new and more sophisticated tests were run” (p.11) and more items were transferred from one subscale to another to create the current version of the CCAI. Given the description of this process in the manual, it is assumed that the same dataset was analyzed for each revision. No cross-validation studies were presented. Limited information regarding the psychometric properties of the CCAI, presumably based on the norming sample data, is presented in the manual. The reliability of the subscale scores from the current version of the instrument ranged from .68 to .90 indicating moderate to high internal consistency. However, limited validity evidence is presented in the manual. Specifically, the development of the CCAI (e.g., use of experts, review of the literature) is presented as content validity evidence. The authors then present a principal components analysis of the items suggesting that the structure weights provide strong construct validity evidence. However, it must be noted that several of the items representing each subscale appeared to be functioning poorly, as they either had Cross-Cultural Adaptability 4 low correlations with the factor they correspond to or high correlations with other factors. In addition, and most importantly, at two points during the development process items that were created to represent one subscale were subsequently reassigned to another subscale. Again, no cross-validation studies supporting these changes were presented. The authors fail to address any of these problems (i.e. cross loadings, reassignment of items, lack of cross-validation) in the manual. In our opinion, this limited and weak validity evidence offers little support pertaining to the development of a measure that represents these constructs. After a review of the literature, only one additional study could be found that examined the structure of the CCAI. However, instead of examining the original fourfactor structure, a 3-factor structure using only 37 of the 50 items was explored using a sample of expatriates (Gelles, 1996). Because the analysis did not examine the proposed four-factor structure or use the entire set of items, it remains unclear if the original dimensionality of the instrument, as stated by Kelley and Meyers (1995a) is actually supported by empirical evidence. Purpose of the current study The CCAI has been used to examine the relationship between cross-cultural adaptability and intercultural experiences such as travel or study abroad programs (Erwin & Coleman, 1998; Kitsantas & Meyers, 2001). In addition, the CCAI has been used to determine the strength of hypothesized predictor variables of cross-cultural adaptability such as impression management (Montagliani & Giacalone, 1998) and general personality characteristics (e.g. temperament, problem-solving abilities; Lui, 1999). Predominately, research with the CCAI has focused on measuring the effectiveness of cultural training programs with law enforcement officers (Cornett-DeVito & McGlone, 2000), teachers (Remmert, 1993), business professionals (Goldstein & Smith, 1999), graduate students (Fukasawa, 1990) and medical professionals (Majumdar, Keystone & Cuttress, 1999). With the importance of accountability today in regard to educational programs, including study abroad programs, practitioners need quality instruments that can assess the success or development of participants in their programs. To ensure the quality of the instruments being used, appropriate validity evidence must be gathered. Despite the numerous studies that have used the CCAI to examine the crosscultural adaptability of various populations, no studies could be found that clearly established the hypothesized four-factor structure. As noted above, the evidence presented in the manual is not sufficient to claim that the CCAI is psychometrically stable or has any construct validity evidence. Therefore, the purpose of the present study was to examine the characteristics of this instrument, in particular the replicability of the fourfactor structure. Method Participants The participants for this study consisted of a random sample of 725 sophomores from a mid-sized, mid-Atlantic university. The participants were administered the CCAI along with several examinations as part of the University’s Assessment program. A total of 709 students completed the CCAI and had the following characteristics: 57% female, 83% Caucasian, 4% African American, 4% Asian, 1% Hispanic, 1% Native American (7% ethnicity not available), average age of 20. Cross-Cultural Adaptability 5 Procedure Students were assigned to classrooms on the designated assessment day and spent approximately 2.5 hours completing assessments in various subject areas. The CCAI was administered when student had completed approximately half of their assessments. Trained proctors administered the instruments. Specifically, the proctors distributed the instruments and read instructions aloud before the students began responding. Students were instructed to read each statement carefully and choose the response that best described them at that point in time. They were allowed 30 minutes to complete the 50 items of the CCAI. Instruments The current version of the CCAI, which was administered for this study, consists of 50 items (Kelley & Meyers, 1995b). Examinees are asked to respond to each item using a scale from 1 (definitely true) to 6 (definitely not true). To compute the four subscale scores, nine of the 50 items on the CCAI have to be reverse coded. A high score on a particular subscale indicates a high level of that attribute. In addition to conducting group comparisons using the subscale scores, the authors note that the four subscale scores can be compared for each person. This type of comparison allows assessment of one’s strengths and weaknesses and can indicate areas where improvements are needed. As noted above, the authors proposed that cross-cultural adaptability has four dimensions. The first construct, emotional resilience, is represented by a subscale consisting of 18 items. When individuals find themselves in a new culture they often experience negative emotional reactions to their situation. Therefore, an important component to being able to adapt to a new environment is the ability to deal with these emotions and still maintain a positive outlook on one’s situation, in other words, to have emotional resilience. The second subscale was created to reflect flexibility and openness. One of the most common components of cross-cultural adaptability is an individual’s capability to posses a non-judgmental attitude and to have an open mind when considering the thoughts and beliefs of others. This 15-item subscale was created to reflect this ability to be broad-minded and open towards others. The third subscale on the CCAI was created to represent perceptual acuity, an individual’s cultural empathy, or “the skill to understand the logic and coherence of other cultures and the restraint to avoid negative attributions based on perceived difference based on one’s own and others’ behavior.” (Dinges, 1983, as cited in Kelly & Meyers, 1995a). This subscale consists of 10 items on the CCAI. The fourth and final subscale of the CCAI is labeled personal autonomy. This refers to an individual’s ability to posses a strong personal identity as well as maintain this identity when placed in a new culture and not feel like they must abandon their personal beliefs to fit in. This subscale consists of 7 items on the CCAI. Results Descriptive Statistics The means and standard deviations for the four subscales are presented in Table 1. Before assessing the factor structure of the responses, the data was examined for normality using PRELIS 2.51 (Jöreskog & Sörbom, 1996). To ensure univariate Cross-Cultural Adaptability 6 normality, Kline (1998) suggests cutoff of absolute values of 3.0 and 8.0 for skewness and kurtosis respectively. The skew of the 50 CCAI items ranged from -2.303 to 1.054 while the values for kurtosis ranged from -.0143 to 7.285 indicating that the responses were fairly normally distributed. In addition, multivariate kurtosis equaled 1.37. While there is no standard cutoff for this index, Bentler (1998) recommends that multivariate normality can be assumed if this value is less than 3. Confirmatory Factor Analysis The raw data was used to create a covariance matrix, which was analyzed using LISREL 8.51 (Jöreskog & Sörbom, 1993). Given the adequate distribution of the responses, maximum likelihood (ML) estimation was used to estimate the model’s parameters and fit indices. This estimation method was used because ML has been found to produce more accurate fit indices and less biased parameters than generalized least squares (GLS) estimation (Olsson, Foss, Troye, & Howell, 2000; Olsson, Troye, & Howell, 1999). The fit of the hypothesized four-factor model was assessed by examining several fit indices. Inadequate fit can indicate complex or simple model misspecification. Simple model misspecification occurs when the covariances between the factors are misspecified while complex model misspecification occurs when one has incorrectly represented the relationship between an item and the factors (Hu & Bentler, 1998). Three absolute fit indices were used to identify model misspecification: the minimum fit function chisquare, the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). The chi-square statistic (χ ) assesses the difference between the sample covariance matrix and the implied covariance matrix from the hypothesized model (Fan, Thompson & Wang, 1999). A non-significant χ indicates adequate model fit. As the sample size increases, the sensitivity of the χ test increases, potentially resulting in small differences causing misfit (Hu and Bentler, 1995). For this reason, additional absolute fit indices were examined. The RMSEA is moderately sensitive to simple model misspecification and very sensitive to complex model misspecification. Hu and Bentler (1998) recommend a cutoff of .06 or less for a wellfitting model. The SRMR is very sensitive to simple model misspecification and moderately sensitive to complex model misspecification. Hu and Bentler (1998) recommend a cutoff of .08 or less for good fit. In addition, one incremental fit index, the comparative fit index (CFI) was examined. The CFI is sensitive to both simple and complex model misspecification. Hu and Bentler (1998, 1999) recommend a cutoff of .95 or above for good model fit. Overall, the fit of the four-factor model was very poor. Specifically, the chisquare statistic (χ = 5381.5, p < .000), the RMSEA (.082), and the CFI (.70) all indicated model misfit. The SRMR (.068) was below the .08 cutoff, indicating adequate fit. As noted above, the SRMR, while highly sensitive to simple model misspecification, is only moderately sensitive to complex model misspecification. Importantly, the CFI and RMSEA are very sensitive to complex model misspecification. Therefore, the lack of fit may be a result of failing to model relationships between items and factors that were hypothesized to be nil. Table 2 displays the standardized factor patterns coefficients and structure coefficients for each item. The factor pattern coefficients indicate the relationship Cross-Cultural Adaptability 7 between each item and it’s corresponding factor. Squaring the pattern coefficient indicates the amount of variance in the item that is explained by the corresponding factor. Therefore a pattern coefficient of .20 would indicate that 4% of the variance in that particular item is explained by it’s corresponding factor. In this particular analysis, the pattern coefficients values range from .06 to .75 indicating that the range of variance explained in all items was 0.36% to 56%. Overall, many of the pattern coefficients appear low and these low values seem to be evenly spread out between the factors. As noted, these low values indicate that the factors are not explaining much variance in the items that serve as indicators for that factor. In addition, it is advantageous to examine the structure coefficients, which estimate the relationship between each item and each factor. These provide additional information because moderate to large relationships could exist despite the fact that the direct paths (pattern coefficients) were fixed to zero. Large structure coefficients that exist between items and factors when the relationship is expected to be nil can be problematic. By examining these values, information about model misspecification could be revealed (Graham, Guthrie & Thompson, 2003; Thompson, 1997). For example, in the CCAI manual, Kelley and Meyers (1995a) state that they used a varimax rotation, which assumes that there are no correlations among the factors. The analysis of this data indicated that there may in fact be large correlations between the factors (.87 .98). In the manual for the CCAI, substantial correlations between the computed subscale scores are reported for the analysis from the norming data (.27-.57). These moderate to high correlations in both studies suggest that there is a problem with discriminant validity. These high correlations suggest that the factors are in fact related and therefore, poor fit may be a result of items on one scale actually being correlated with other subscales (Nunnally & Bernstein, 1994). The effects of these high correlations are evident in the structure coefficients presented in Table 2. Many of the items that have pattern coefficients that were fixed to zero have substantial structure coefficients. In fact, the large structure coefficients are evidence that strong relationships exist between factors and indicators other than their own. Some of these unmodeled relationships (structure coefficients) are almost as strong as the relationship between that item and it’s corresponding factor (pattern coefficients). This coincides with what the fit indices indicated; we have complex model misspecification. Furthermore, the modification indices that were suggested by LISREL to improve the fit of this model were examined but were too extensive to be interpreted. Exploratory Factor Analysis Due to the poor fit of the original model, an EFA was run using SPSS 11.0 in an attempt to uncover the structure underlying the responses. Principal axis factoring was used for this analysis. Although this was not the method that the authors used to assess the structure (principal components was used), it was chosen because it takes measurement error into account (Benson & Nasser, 1998). Initially, four factors were extracted in an attempt to replicate the original structure; oblimin rotation was used in order to examine the relationships among the factors. The first factor extracted explained 27.6% of the variance while the other 3 factors explained 3.8% to 5.5% resulting in a total of 41.19% of the total variance explained. The structure coefficients are presented in Table 3. When looking at this table, one can see that 17 items correlate the highest with Cross-Cultural Adaptability 8 the first factor, 9 correlate the highest with the second factor, 20 items correlate the highest with the third factor, and 4 items correlate the highest with the fourth factor. Based on the eigenvalues, variance explained, and structure coefficients, it is apparent that no clear structure was formed here and the original model could not be replicated. Discussion This research was designed to examine the replicability of the four-factor structure of the Cross-Cultural Adaptability Inventory proposed by Kelley and Meyers (1995a). Unfortunately, the four-factor model did not fit this data as demonstrated by the results from the CFA. An exploratory factor analysis failed to produce a structure that was interpretable. There are several possible reasons why the four-factor model did not fit this data set. First, as indicated by the high inter-factor correlations and the structure coefficients, the latent variables seem to be closely related. Therefore, unmodeled relationships (cross loadings) most likely caused the fit indices to indicate that the model had complex misspecification. Second, as shown by the confirmatory factor analysis, several of the CCAI items had very little variance explained by their corresponding factor. It could be that these items are not functioning properly, are on the wrong subscale, or are just not related to the constructs that the CCAI was designed to measure. These are just two of the possible reasons why this model did not fit our data set. While we may be unsure of the exact reason for misfit, we can be sure that this four-factor model does not fit with our data and cross-cultural adaptability appears to be a construct that is not measurable by these items and/or this structure. In conclusion, it is our recommendation that this instrument not be used to assess the cross-cultural adaptability of any population until it has been studied further. This scale appears to need major revision including item analysis and structural changes. Modifications need to be made based on theoretical research and supported by statistical evidence. Cross-Cultural Adaptability 9 ReferencesBenson, J., & Nassar, F. (1998). On the use of factor analysis as a research tool. Journalof Vocational Education Research, 23(1), 13-33.Bentler, P. (1998, March 10). Kurtosis, residuals, fit indices. Message posted toSEMNET discussion list, archived at http://bama.ua.edu/cgi-bin/wa?A2=ind9803&L;=semnet&T;=0&F;=&S;=&P;=14689Cornett-DeVito, M., & McGlone, E. (2000). Multicultural communication training forlaw enforcement officers: A case study. Criminal Justice Policy Review, 11(3),234-253.Dinges, N. (1983). Intercultural competence. In D. Landis & R. Brislin (Eds.), Handbookof intercultural training (Vol. 1, pp.176-202). New York: Pergamon Press.Erwin, T. D., & Coleman, P. K. (1998). The influence of intercultural experiences andsecond language proficiency on college students' cross-cultural adaptability.International Education Journal, 28, 5-25.Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods,and model specification on structural equation modeling fit indexes. StructuralEquation Modeling, 6(1), 56-83.Fukasawa, L. (1990). The effect of multicultural counseling training on multiculturalsensitivity of graduate students. (Doctoral dissertation, Indiana StateUniversity). Dissertation Abstracts International, 51, 05A.Gelles, R. (1996). Expatriate adjustment and performance, and spouse adjustment.(Doctoral dissertation, United States International University). DissertationAbstracts International, 57, 2.Graham, J., Guthrie, A., & Thompson, B. (2003). Consequences of not interpretingstructure coefficients in published CFA research: A reminder. StructuralEquation Modeling, 10(1), 142-153.Goldstein, D., & Smith, D. (1999). The analysis of the effects of experiential training onsojourners’ cross-cultural adaptability. International Journal of InterculturalRelations, 23, 157-173.Hu, L., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structuralequation modeling: Concepts, issues, and applications (pp. 76-99). ThousandOaks, CA: Sage.Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivityto underparameterized model misspecification. Psychological Methods, 3, 424-453.Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structureanalysis: Conventional criteria versus new alternatives. Structural EquationModeling, 6, 1-55.Jöreskog, K., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with theSIMPLIS command language. Chicago: Scientific Software International.Jöreskog, K., & Sörbom, D. (1996). PRELIS 2: User’s reference guide. Chicago:Scientific Software International.Kelley, C., & Meyers, J. (1995a). The Cross-Cultural Adaptability Inventory (manual).Minneapolis, MN: National Computer Systems.Kelley, C., & Meyers, J. (1995b). The Cross-Cultural Adaptability Inventory.Minneapolis, MN: National Computer Systems. Cross-Cultural Adaptability 10 Kelley, C., & Meyers, J. (1999). The Cross-Cultural Adaptability Inventory. In S. Fowler& M. Mumford (Eds.), Intercultural Sourcebook: Cross-Cultural TrainingMethods (Vol. 2, pp 53-70). Yarmouth: Intercultural Press.Kitsantas, A., & Meyers, J. (2001, March). Studying Abroad: Does it enhance collegestudent Cross-Cultural awareness? Paper presented at the combined AnnualMeeting of the San Diego State University and the U.S. Department of EducationCenters for International Business Education and Research, San Diego, CA.Kline, R. B. (1998). Principles and Practice of Structural Equation Modeling. NewYork: Guilford.Lui, G. (1999). Personal characteristics associated with the Cross-Cultural AdaptabilityInventory for Hong Kong Chinese missionaries. (Doctoral dissertation, FullerTheological Seminary). Dissertation Abstracts International, 60, 4.Majumdar, B., Keystone, J., & Cuttress, L. (1999). Cultural sensitivity training amongforeign medical graduates. Medical Education, 33, 177-184.Montagliani, A., & Giacalone, R. (1998). Impression management in cross-culturaladaptation. Journal of Social Psychology, 138, 598-608.Nunnally, J., & Bernstein, I. (1994). Psychometric Theory (3 ed). New York:McGraw-Hill.Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000). The performance of ML,GLS, & WLS estimation in structural equation modeling under conditions ofmisspecification and nonnormality. Structural Equation Modeling, 7, 557-595.Olsson, U. H., Troye, S. V., & Howell, R. D. (1999). Theoretic fit and empirical fit: Theperformance of maximum likelihood versus generalized least squares estimationin structural equation modeling. Multivariate Behavioral Research, 34, 31-58.Remmert, A. (1993). The impact of multicultural in-service education on the cross-cultural adaptability of public school teachers. (Doctoral dissertation,Florida State University). Dissertation Abstracts International, 54, 12A.Thompson, B. (1997). The importance of structure coefficients in structural equationmodeling confirmatory factor analysis. Educational and PsychologicalMeasurement, 57(1), 5-19. Cross-Cultural Adaptability 11Table 1Descriptive statistics for the four subscales of the CCAI Mean Standard Deviation Emotional resilience 46.528.47 Perceptual acuity 23.835.41Personal autonomy 15.233.74Flexibility/Openness 45.916.00 Note: N=709. The subscales scores have the following range:Emotional Resilience: 18-108Perceptual Acuity: 10-60Personal Autonomy: 7-42Flexibility/Openness: 15-90 Cross-Cultural Adaptability 12Table 2Standardized Factor Pattern and Structure Coefficients from the Confirmatory FactorAnalysis of the Four-Factor Model Item EmotionalResilienceFlexibility/ OpennessPerceptual